The disclosure relates generally to a method and apparatus for reconstructing video frames that include missing pixels as a result of video stabilization techniques.
Videos captured using video cameras which are unstable, either as a result of movement of a human hand supporting the camera or movement of another supporting structure such as a helicopter or other mobile platform, typically suffer from significant image motion or shaking. The undesirable motion may result from camera displacement (i.e., vertical and/or lateral motion), rotation, and/or zooming. Mechanical methods of camera stabilization cannot entirely eliminate the undesirable motion, and are typically very expensive to employ. Thus, software implemented video stabilization or motion compensation techniques are frequently used in video playback and/or processing devices wherein the frames of video are transformed to eliminate unintended image movement due to camera displacement, rotation and/or zooming. Unfortunately, as a result of this transformation, areas in the transformed frames may include missing pixels, often around the borders of the frames for reasons described below.
Various approaches exist for reconstructing the missing pixels in video frames that have been transformed to compensate for undesired camera motion. One approach includes frame zooming or cropping, wherein the transformed frames are reduced in size such that the missing pixels in border areas of the frames are eliminated from the outputted frames. Using this approach, however, also results in a loss of content along the border areas of the frames, as well as a reduction in image resolution.
In another approach, sometimes referred to as pixel mirroring or border extension, missing pixels along a border of a frame are filled in using adjacent pixels in the image or by repeating pixels at the image border to the edge of the frame. While blank areas in the transformed frames are filled using this approach, the content used for the missing pixels is not representative of the actual image, and visible artifacts are typically produced.
In yet another approach, image mosaics are constructed by accumulating neighboring frames and stitching areas from those frames over the missing areas in a current frame. This mosaicing or image stitching produces good results for static scenes, but, due to the geometric transformation model typically employed, generates visible artifacts such as unnatural discontinuities when used for dynamic scenes.
Finally, another approach called exhaustive searching requires storage of a long history of past frames and includes a comparison of all pixels in a current frame to all of the pixels in preceding frames to substitute pixels in prior frames for missing pixels in the current frame. This approach can provide satisfactory image reconstruction, but it requires significant storage capacity, very high processing power and relatively long processing time. Consequently, the exhaustive search approach is cost prohibitive for some applications from a memory and processing standpoint, and cannot be used to perform real-time video reconstruction.
Accordingly, there exists a need for an improved method and apparatus for reconstructing motion compensated video frames that addresses one or more of the above-noted drawbacks.